System and methods for eye tracking are known in the art. U.S. Pat. No. 5,583,795 to Smyth entitled “Apparatus for Measuring Eye Gaze and Fixation Duration, and Method Therefore”, is directed to a device and method for measuring eye gaze and fixation duration. An electronic video display array provides controlled illumination of the eye of a user. The initial flares in display pixel brightness, generated during the electronic refresh sweep of the display, creates a sequence of point light sources. The reflections from the cornea and internal eye components reach the array of light sensors through display optics. The sensor system comprises an array of phototransistors (amplifiers), comparators, and an encoder and latch, for isolating the transistor with maximal response to the corneal reflex from the instantaneously excited light source. The corresponding video display pixel element is determined by the display refresh order, indexed by the raster scan clock pulse count. This encoded address is written by the processor onto digital memory, eventually creating a table of array matrix address for phototransistor sensor and light source pairings, accessible by digital computer routines.
The digital computer runs several routines to compute: visual line of sight, workspace line of sight, fixation status, and gaze point for each eye. The visual line of sight routine generates a three dimensional model of the eye. Utilizing the stored reflection data, the routine obtains the corneal surface reflection points, the corneal center and the pupil center, for computing of the optical origin and axis. The routine then smoothes reflection point locations using a clustering algorithm, and analyzes this data to determine the median axes. Finally, the routine computes the viewing origin and axis from the optical and median axes.
U.S. Pat. No. 6,120,461 to Smyth entitled “Apparatus for Tracking the Human Eye with a Retinal Scanning Display, and Method Therefore”, is directed to a device and method for measuring eye gaze and fixation duration, wherein a retinal scanning display creates sequential sources of light. The directed light sweeps the retina in discrete steps, illuminating adjacent portions of the retina in a point-wise manner and creating a succession of reflection points. The reflections are detected by an active-pixel image sensor array composed of CMOS substrate. The sensor is integrated with a comparator array and an element address encoder and latch, both clocked by the raster scan pulse of the display driver. This determines the sensor element reaching maximum intensity or activated from a particular sequential light source reflection off the cornea. Over a refresh cycle, the corneal surface is mapped to a data table through pairings of sensor activations and corneal reflections. A CCD device (or alternatively a memory cache) is activated by the sensor array to record diffused reflections, or sensor pixels with intensities less than that of the direct reflections, as determined by the comparator array. These parameters are also used in subsequent data analysis.
After each cycle, the image processor, comprised of a stack of VLSI circuit arrays, generates a three dimensional image of the human eye, expanded to include more features and inner structures of the eye. The image processor computes a model of the cornea and optical locations for isolated image features. The optical origin and the optical and median axes are computed from internal eye features, including corneal optic center and axes, corneal surface center and axes, pupil optic center, pupil image orientation, capillary network of retinal fundus and iris pattern. The viewing origin and axis is obtained from the optical origin and axis and median axes.
U.S. Pat. No. 5,331,149 to Spitzer et al, entitled “Eye tracking system having an array of photodetectors aligned respectively with an array of pixels”, is directed to an eye tracking apparatus. A flat panel display projects an image, through a photodetector array and onto a viewing screen, which is viewed by the user. Each pixel in the display is aligned with a corresponding photodetector. The light rays which are reflected from a display pixel to the fovea of the eye, are reflected back from the eye along the same optical path. The photodetector identifies the pixel from which the light ray emanated by generating a voltage signal in the array unit nearest the returned light ray. The corresponding portion of the display represents the line of sight of the user. A cursor is projected on the screen at the line of sight location to provide feedback.
In order to prevent interference from outside light, a bandpass filter, placed over the array, blocks out all wavelengths but that of the cursor. A corresponding band rejection filter is placed on the outside of the viewing screen. Alternatively, a pair of polarizers may be used instead of filters. A light polarizer is placed over the detector array, in conjunction with a ninety degree-crossed polarizer over the viewing screen. Further alternatives include using infrared light from the display, or blinking the cursor image allowing the computer to eliminate background light.
U.S. Pat. No. 6,433,760 to Vaissie et al. entitled “Head Mounted Display with Eye tracking Compatibility”, is directed to a display and eye tracking system. The system includes a CCD camera, an LCD, a computer, an imaging system, at least one LED, a hot mirror and a cold mirror.
The LEDs emit 900 nm light. One of the mirrors reflects the light onto the cornea of the eye. The reflected infra-red beam from the cornea of the eye strikes the hot mirror. The hot mirror directs the reflected infra-red beam through the imaging system. The beam then passes through the cold mirror and is focused onto the CCD camera. The computer processes the beam to determine the sight direction of the user.
The LCD screen receives visual information from the computer. The imaging system displays the images from the LCD screen. The hot mirror reflects the rays from the LCD screen onto the cornea.
U.S. Pat. No. 6,396,461 to Lewis et al. entitled “Personal Display with Vision Tracking”, is directed to a display apparatus. The apparatus includes control electronics, a light source, a scanning assembly and imaging optics. The imaging optics is formed from curved, partially transmissive mirrors. The mirrors receive light from a background and from the scanning assembly. The mirrors combine the light received from these sources to produce a combined image to the eye of a viewer.
The imaging optics redirects and magnifies scanned light from the scanning assembly toward the eye. The scanned light passes through the pupil of the eye, and strikes the retina of the eye to produce a virtual image. Background light passes through the mirrors and the pupil to the retina of the user, to produce a “real” image.
The apparatus further includes an infrared light source, positioned adjacent to the light source, and an optical detector. A common substrate carries the infrared light source and the light source. The imaging optics receives a locator beam from the infrared light source. The imaging optics redirect light, reflected from the eye, toward the optical detector. The detector data are input to an electronic controller. The controller accesses a look up table in a memory device to retrieve positioning data indicating a correction for the light source. The controller activates X and Y drivers to provide voltages to respective piezoelectric positioners, coupled to the substrate, for correcting the positions of the light sources.
U.S. Pat. No. 6,381,339 to Brown et al. entitled “Image System Evaluation Method and Apparatus Using Eye Motion Tracking”, is directed to an eye tracking apparatus for evaluating different image systems.
The eye tracking apparatus consists of a video-based, infrared illuminated, headband mounted eye tracking technique. A collimated diode emitting infrared light illuminates the eye. A monochrome CCD camera is aligned coaxially with the diode. The camera captures “bright-pupil” reflection from the retina of the subject and the first surface reflection of the cornea (“first Purkinje image”). An eye tracker control unit digitizes the camera images and thresholds the image at two levels in real-time.
The first threshold level isolates pixels within the bright pupil, and the second threshold level isolates those pixels that are within the corneal reflection.
A lab computer then computes the centroid of the pupil and the first Purkinje image. The eye-in-head position is calculated based on the relative location of the two centroids when both items are available in the camera image, in order to make the system less sensitive to movement of the tracker with respect to the head.
A magnetic field head tracker monitors head position and orientation in real time. The head tracker comprises a transmitter unit mounted above the head of the subject, which contains three orthogonal coils that are energized in turn. A receiver unit contains three orthogonal antennae coils that pick up the corresponding signals from the transmitter. The head position and orientation points are determined from the absolute and relative strengths of the transmitter/receiver coil pairs.
The gaze position is then calculated, using eye-in-head and head position/orientation data, in the form of the intersection of the line-of-sight with the working plane. The eye tracking apparatus provides a digital data stream containing eye-in-head, head orientation and position, and gaze intercept information. In addition a camera present on the headband provides a video record of the scene from the perspective of the subject, also indicating the same positional data.
U.S. Pat. No. 6,158,866 to Gulli et al. entitled “Optical System Combining Image Presentation and Eye Analysis”, is directed to an image presentation system. The image presentation portion of the system comprises an image generator and an optical transmission channel. The image generator sends images to a display screen. An optical transmission channel passes the images from the display to the eye of the use. The channel includes a collimating device that projects the screen image to be perceived by the user as located at an infinite distance, or at a finite distance if desired. The channel also includes a combiner allowing the user to perceive a superimposed image in visible light.
The eye tracking portion comprises an illuminating system and image detection system. The illuminating system consists of a light source and optical transmitter which illuminates the retina at the rear inner surface of the eye of the user. The illuminating light wave propagates through a bundle of optical fibers before reaching the eye. A scanning system at the light source allows for selective illumination of different fiber groups and scans the retina. The reflected light follows the inverse path of the illuminating light, passing through the optical fibers to the scanning system. A semi-reflecting mirror reflects the light to the image detection system. The image detection system consists of a detector that detects the intensity of reflected light, and a device that generates a retinal image of the eye.
Conventional helmet mounted display systems use the line of sight of the helmet in order to aim at a target. Such systems utilize a position and orientation sensor, mounted on the helmet, in order to define the helmet line of sight. Thus, a pilot needs to move his head and helmet (i.e., using the neck), in order to aim at a target.
U.S. Pat. No. 6,667,694 B2 issued to Ben-Ari et al., and entitled “Gaze-Actuated Information System” is directed to a gaze-actuated information system for generating an audio output for a pilot of an aircraft, according to an eye gaze direction of the pilot and a reference direction. The gaze-actuated information system includes a helmet mounted system, a cockpit mounted system, a weapon system unit and a weapon system.
The helmet mounted system includes an eye tracking system, a transmitter and a first power supply. The cockpit mounted system includes a first transceiver, a processor, an audio system, a helmet position system and a second power supply. The weapon system unit includes a second transceiver, a control interface and a third power supply. The weapon system includes a seeker and a launcher. The processor includes a direction correlation system. The eye tracking system, the transmitter, the first power supply, the first transceiver, the processor and the helmet position system form a gaze-direction determining system.
The eye tracking system derives the eye gaze direction of an eye of the pilot relative to a helmet of the pilot. The helmet position system derives the position of the helmet within a cockpit of the aircraft. The processor derives the eye gaze direction of the eye of the pilot relative to a frame of reference moving with the cockpit, according to the eye gaze direction relative to the helmet, and according to the position of the helmet. The weapon system unit relays seeker direction information from the weapon system to the cockpit mounted system.
When the pilot looks at a target, the seeker is locked to the eye gaze direction of the pilot. The pilot brings the seeker into alignment with the target by looking toward the target, and designates the target by depressing a control button. Depressing the control button releases the seeker from the eye gaze direction, and allows the seeker to track the target. At this stage the audio system generates a first audible signal to indicate to the pilot that the seeker has locked on to the target. Before firing a missile toward the target, the pilot verifies that the seeker has locked on to the correct target.
For performing this verification, the direction correlation system compares the eye gaze direction relative to the frame of reference with the target direction relative to the frame of reference. If the eye gaze direction and the target direction are equal within a given degree of accuracy, then the direction correlation system determines that the pilot is currently looking at the target which is being tracked, and the audio system generates a predefined audible signal.
The direction correlation system compares the eye gaze direction relative to the frame of reference with a reference direction relative to the frame of reference. The reference direction is chosen to correspond to a region of the field of view of the pilot, with which certain information is associated. If the eye gaze direction and the reference direction are equal within a given degree of accuracy, then the audio system generates audio output to the pilot indicative of the information associated with that reference direction.